
IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 
BEFORE THE BOARD OF PATENT APPEALS AND INTERFERENCES 



In re Application of Krishnamachari, S. Atty. Docket No.: PHA 23-431 A 

Serial No.: 09/934,962 Group Art Unit: 2623 

Filed: 8/22/2001 Examiner: Wu, J. 

Title: COLOR QUANTIZATION AND SIMILARITY MEASURE FOR CONTENT 

BASED IMAGE RETRIEVAL 

RECEIVED 

APPELLANT'S BRIEF ON APPEAL UNDER 37 C.F.R. S 1,192 

DEC 1 0 2002 

Honorable Commissioner of Patents and Trademarks _^ 

Washington, D.c. 2023 1 Technology Center 2800 

Sir: 

This is an appeal from the decision of the Examiner dated 8 July 2002, finally 
rejecting claims 1-20 of the subject application. 



L REAL PARTY IN INTEREST 

The above-identified application is assigned, in its entirety, to Philips Electronics 
North America Corporation, a company organized under the laws of the State of 
Delaware. 

II. RELATED APPEALS AND INTERFERENCES 

Appellant is not aware of any co-pending appeal or interference which will 
directly affect or be directly affected by or have any bearing on the Board's decision in 
the pending appeal. 



IIL STATUS OF CLAIMS 

Claims 1-20 are pending in the application. Claims 1-20 stand rejected by the 
Examiner under 35 U.S.C. 102(e). 



IV, STATUS OF AMENDMENTS 

An amendment is filed concurrently. No other amendments were filed subsequent S 

o 

to the final rejection in the Office Action dated 8 July 2002. § 
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V. SUMMARY OF THE INVENTION 

The invention comprises a method and system for characterizing images, and a 
method and system for comparing images based on this characterization. 

The characterization of the image is based on a pluraHty of measures that are 
proportional to the frequency of occurrence of a plurality of colors. The image is 
partitioned into a plurality of partitions. Each partition is characterized by the proportion 
of colors within the partition, such as: 10% red, 30% green, 35% white, 2% black, 5% 
yellow, 12% cyan, and so on. The number of different colors used for the characterization 
is preferably fixed, and based on the expected or observed distribution of colors among a 
collection of images (Applicant's page 5, line 9 through page 6, line 2). The 
characterization of the image is a representation of the proportion of occurrences of each 
color within each partition of the image. For example, if the number of different colors is 
64, the characterization of the image comprises the proportion of each of these 64 
different colors in each of the partitions. (Applicant's page 7, lines 3-12). 

To perform a comparison of images, two images are each characterized as 
detailed above, providing a characterization representing the proportion of occurrences of 
each color within each partition of each image. These characterizations are compared by 
comparing the proportions of similar colors in each corresponding partition of the 
images. Preferably, to provide for an efficient comparison, only a predefined number of 
the most popular colors in each partition are compared. If, for example, the four most 
popular colors in a partition of the first image are red, green, yellow, and black, the four 
colors in a corresponding partition of the second image that are most similar to red, 
green, yellow, and black are selected for comparison. The comparison of these four 
colors includes a comparison of the proportions of each of these most-similar colors in 
each partition, as well as a measure of the similarity between the colors. Preferably, the 
similarity-measure is based on the number of occurrences of similar colors, weighted by 
the degree of similarity between the colors. (Applicant's page 7, line 17 through page 8, 
line 1 1.) By comparing the proportions of similar-color pixels in each partition of an 
image, the comparison process of this invention is less dependent upon such factors as 
lighting and shading (Applicant's page 8, lines 1 1-18). 
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VL ISSUES 

Are claims 1-5 and 1 1-15 patentable under 35 U.S.C. 102(e) in view of Sato et al. 
(USP 6,181,818, hereinafter Sato)? 

Are claims 6-10 and 16-20 patentable under 35 U.S.C. 102(e) in view of Sato? 

VIL GROUPING OF CLAIMS 

Claims 1-5 and 1 1-15 stand or fall together. Claims 6-10 and 16-20 stand or fall 
together. 

VIII, ARGUMENT 

Claims 1-5 and 1 1-15 are patentably distinct from claims 6-10 and 16-20 because 
claims 1-5 and 11-15 claim a method and system for characterizing images, whereas 
claims 6-10 and 16-20 claim a method and system for comparing two or more images. 

Are claims 1-5 and 11-15 patentable under 35 U.S.C. 102(e) in view of Sato? 

Independent claims 1 and 15 each claim a characterization of the image based on 
a plurality of measures that are proportional to the frequency of occurrence of a plurality 
of colors. 

Sato teaches creating a histogram of the frequency of occurrences of a plurality of 
colors in each partition, but does not create a characterization measure that is 
proportional to these frequencies of occurrences. Sato uses the histogram of frequency of 
occurrences to compare each frequency to a threshold value, T(R), to determine whether 
to include the particular partition in a list of partitions that contain the color. Sato's FIG. 
44, steps S174-S175 illustrates the comparison to the threshold T(R), and FIG. 45 
illustrates an entry in a list for a given partition size. The array illustrated in FIG. 45 
corresponds to the characterization of an image as taught by Sato. Sato refers to this 
characterization as an "index table" of the image, and specifically states: 

"When the operation shown in FIG. 44 is repeated for all the resolutions (block 
sizes) R and all the blocks j, index information to images and regions having a 
predetermined frequency or higher of a specific color is stored in the index table 
in units of blocks'' (Sato, column 26, lines 13-19.) 
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As specifically taught by Sato, the characterization of the image is a table that 
lists the blocks (partitions) that contain each color above a given threshold amount. If the 
threshold amount is 10%, for example, a block with a proportion of 15% of a given color 
will be included in the table at the entry for that color, in like marmer, a block with a 
proportion of 80% of the color will be included, as will a block with 45% and so on. As 
illustrated in FIG. 45, and as specifically recited above, the entry in the table is the name 
of the block, only. Sato does not teach the inclusion of the particular percentage of the 
color of each block in the index table. That is, once the identification of the above 
example blocks are entered in the table, it is impossible to distinguish which if any of 
these blocks had a larger or smaller proportion of the color than any other block, because 
Sato does not store the proportion in the table. In the specific example illustrated in FIG. 
45 of Sato, for example, it is impossible to determine whether "(IMAGE 1, REGION 1)" 
has more or fewer pixels of color CI than "(IMAGE 2, REGION 3)". The only thing that 
is known about these two regions is that each have more than the threshold number of 
pixels of color CI. 

Sato's index table contains names of blocks only, and does not contain a measure 
that is proportional to the firequency of occurrences of a color, and, as such, does not 
anticipate the Applicant's invention as specifically claimed in independent claims 1 and 
11, upon which claims 2-5 and 12-15 depend. 
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Are claims 6-10 and 16-20 patentable under 35 U.S.C, 102(e) in view of Sato? 

The Applicant specifically claims a method and system that compares images by 
comparing the frequency of occurrence of select sets of colors in each corresponding 
partition of two images. 

Sato teaches the comparison of two images by a comparison of regions of an input 
image to an index table. The size of each region of the input image determines which 
column (block size) of the index table is used, and the color/hue of the region identifies 
the row of the table. The output of Sato's comparison is the Ust of candidate images found 
in the index table at the determined row and column, in the form of "candidate image 
number (image id) and search block number (region id)". (Sato, column 26, lines 19-32). 
That is, if the predominant color of the input region is CI, and the size of the input region 
corresponds to a block size of 16, Sato's search of the index table in FIG. 45 will produce 
two candidate images, and will identify "(IMAGE 1, REGION 1)" and "(IMAGE 2, 
REGION 3)" as the corresponding similar-size-and-color regions. As noted above, and as 
illustrated in FIG. 45, the index table does not contain the proportions of color CI in each 
of these regions, and thus the comparison process of Sato cannot compare the proportion 
of color in the input region to this unknown proportion in each of the identified regions. 

As specifically noted by Sato, the index table is used as a "primary screening 
operation", and other techniques are used to perform comparisons that are more detailed. 
(Sato, column 27, lines 15-20). The identification of candidate regions that contain at 
least a threshold amount of the input color, regardless of the particular proportion of the 
color in each region, serves to effect this "primary screening operation" with minimal 
computational complexity. 

Because Sato does not store the firequency of occurrence of each color in each 
partition in the index table, Sato cannot be said to effect a comparison of the frequencies 
of occurrences of select colors in corresponding partitions of two images, as specifically 
claimed in each independent claim 6 and 16, upon which claims 7-10 and 17-20 depend. 
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CONCLUSIONS 

Because Sato stores only an identifier of each region that contains a threshold 
amount of each color, and does not store a measure of the proportion of each color in 
each region, the Applicant respectfully requests that the Examiner's rejection of claims 1- 
5 and 11-15 under 35 U.S.C. 102(e) be reversed by the Board, and the claims be allowed 
to pass to issue. 

Because Sato stores only an identifier of each region that contains a threshold 
amount of each color, and does not store the frequency of occurrence of each color in 
each region for subsequent comparisons, the Applicant respectfully requests that the 
Examiner's rejection of claims 6-10 and 16-20 under 35 U.S.C. 102(e) be reversed by the 
Board, and the claims be allowed to pass to issue. 



ResnectfUlly submitted 





Robert M. McDermott, Attorney 
Registration Number 41 ,508 
804-493-0707 



CERTIFICATE OF MAILING 
It is hereby certifled that this correspondence is being deposited with the United States Postal Service as first-class mail in an 
envelope addressed to: COMMISSIONER OF PATENTS AND TRADMrd?RKS, Wa^ington, D.C. 20231 

On 7 December 2002 




PHA 23,431 A Appeal Brief 



6 December 2002 




APPENDIX 

CLAIMS ON APPEAL (as amended concurrently) 

1. A method for characterizing an image comprising: 

partitioning the image into a plurality of partitions, each partition including a 
plurality of pixels, each pixel having a color, 

determining a frequency of occurrence of each color of the plurality of pixels 
within each partition, and 

creating a characterization of the image that includes a plurality of measures that 
are proportional to the frequency of occurrence of a plurality of colors. 

2. The method of claim 1, further including 

quantizing an encoded color of each pixel to provide the color of each pixel. 

3. The method of claim 2, further including 

identifying a plurality of populous colors, based on the frequency of occurrence of 
each color, and 

the plurality of measures includes proportions of each of the plurality of populous 
colors in each partition. 

4. The method of claim 2, wherein 

quantizing the encoded color includes 

identifying a set of color centers, and 

determining the color of each pixel based upon a color distance between 
the encoded color of each pixel and each of the set of color centers. 

5. The method of claim 1, further including 

identifying a plurality of populous colors, based on the frequency of occurrence of 
each color, and 

the plurality of measures includes proportions of each of the plurality of populous 
colors in each partition. 
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6. A method of comparing a first image to a second image, comprising 

partitioning the first image into a plurality of first partitions, each first partition 
including a plurality of first pixels, each first pixel having a color, 

determining a frequency of occurrence of each color of the plurality of first pixels 
within each first partition, 

partitioning the second image into a plurality of second partitions, each second 
partition including a plurality of second pixels, each second pixel having a color, 

determining a frequency of occurrence of each color of the plurality of second 
pixels within each second partition 

comparing the frequency of occurrence of a select set of colors in each first 
partition with the frequency of occurrence of a corresponding select set of colors in each 
second partition. 

7. The method of claim 6, further including 

quantizing an encoded color of each pixel of the plurality of first pixels to provide 
the color of each pixel of the plurality of first pixels. 

8. The method of claim 7, further including 

identifying a plurality of first populous colors, based on the frequency of 
occurrence of each color of the plurality of first pixels, and 

identifying a plurality of second populous colors, based on the frequency of 
occurrence of each color of the plurality of second pixels; and 

wherein 

the select set of colors in each first partition corresponds to the plurality of first 
populous colors, and 

the corresponding set of colors in each second partition is based upon a color 
difference between each of the plurality of second populous colors and the plurality of 
first populous colors. 
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9. The method of claim 7, wherein 

quantizing the encoded color includes 

identifying a set of color centers, and 

determining the color of each pixel based upon a color distance between 
the encoded color of each pixel and each of the set of color centers. 

10. The method of claim 6, further including 

identifying a plurality of first populous colors, based on the frequency of 
occurrence of each color of the plurality of first pixels, and 

identifying a plurality of second populous colors, based on the frequency of 
occurrence of each color of the plurality of second pixels; and 

wherein 

the select set of colors in each first partition corresponds to the plurality of first 
populous colors, and 

the corresponding set of colors in each second partition is based upon a color 
difference between each of the pluraUty of second populous colors and the plurality of 
first populous colors. 
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1 1 . A system for characterizing an image comprising: 

a partitioner that is configured to partition the image into a plurality of partitions, 
each partition including a plurality of pixels, each pixel having a color, and 

an accumulator that is configured to determine a frequency of occurrence of each 
color of the plurality of pixels within each partition, and 

wherein 

the system is configured to create a characterization of the image that includes a 
plurality of measures that are proportional to the frequency of occurrences of a plurality 
of colors. 

12. The system of claim 11, further including 

a quantizer that is configured to quantize an encoded color of each pixel to 
provide the color of each pixel. 

13. The system of claim 12, wherein 

the plurality of measures are based on the frequency of occurrence of each of a 
plurality of populous colors in each partition. 

14. The system of claim 12, wherein 

the quantizer is configured to quantize the encoded color based upon a color 
distance between the encoded color of each pixel and each of a set of color centers. 

15. The system of claim 11, wherein 

the plurality of measures are based on the frequency of occurrence of each of a 
plurality of populous colors in each partition. 
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16. A system for comparing a first image to a second image, the system comprising: 

a similar color determinator that is configured to determine 

a mapping between a first set of colors of pixels of the first image and a 
second set of colors of pixels of the second image, based on a color distance between 
each of the first set of colors and each of the second set of colors, 

the mapping thereby providing a corresponding color in the second set of 
colors for each color in the first set of colors, and 

a similarity determinator that is configured to determine an image similarity 
measure based on a comparison of a frequency of occurrence of pixels of each of the first 
set of colors and a frequency of occurrence of pixels of each of the corresponding colors 
in the second set of colors. 

17. The system of claim 16, wherein 

the first image is partitioned into a plurality of first partitions, 

the second image is partitioned into a plurality of second partitions, 

the similar color determinator is configured to determine the mapping between the 

first and second sets of colors of pixels for each partition of the plurality of first and 

second partitions, and 

the similarity determinator is configured to determine a plurality of similarity 

measures based on the comparison of the frequencies of occurrence of pixels of each of 

the first and second set of colors for each partition of the plurality of first and second 

partitions, and further includes 

an accumulator that is configured to provide the image similarity measure 

based on a composite of the plurality of similarity measures corresponding to each 

partition of the first and second partitions. 

18. The system of claim 17, wherein 

the similarity determinator is further configured to determine the similarity 
measure based upon the color distances between each of the first set of colors and the 
corresponding color in the second set of colors. 
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19. The system of claim 16, wherein 

the first set of colors of the pixels of the first image is based on a quantization of 
encoded colors of the pixels of the first image. 

20. The system of claim 17, wherein 

the quantization of encoded colors is based on a color distance between the 
encoded color of each pixel and each of a set of color centers. 
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